Skip to content

[Benchmark] Add benchmark functional test framework with Qwen3 test case#1145

Merged
Caozhou1995 merged 22 commits into
flagos-ai:mainfrom
zihugithub:add-benchmark
Mar 19, 2026
Merged

[Benchmark] Add benchmark functional test framework with Qwen3 test case#1145
Caozhou1995 merged 22 commits into
flagos-ai:mainfrom
zihugithub:add-benchmark

Conversation

@zihugithub

@zihugithub zihugithub commented Mar 12, 2026

Copy link
Copy Markdown
Collaborator

PR Category

CICD

PR Types

New Features

PR Description

Add a benchmark functional test framework to the CI/CD pipeline, enabling automated detection of training performance regressions by comparing runtime metrics against baseline gold values.

  • Introduce dedicated benchmark workflow (functional_tests_benchmark.yml) with artifact download retry logic, environment auto-setup, benchmark log parsing, metrics upload to backend, and failure log archiving
  • Add test_benchmark_equal in check_results.py supporting two threshold types (upper_bound for elapsed time, lower_bound for throughput) with configurable tolerance and automatic warmup iteration skipping (first 5 iterations)
  • Add parse_benchmark_output.py to extract metrics from training logs into structured JSON and upload to the metrics backend for tracking
  • Register benchmark as a new task type in the test runner (run_functional_tests.sh), platform config loader (load_platform_config.sh), and platform definition (cuda.yaml)
  • Add Qwen3 tp2_pp2 as the first benchmark test case with gold values for elapsed time per iteration (ms) and throughput per GPU (TFLOP/s/GPU) at 10% tolerance
  • Integrate benchmark tests into all_tests_common.yml and re-enable inference/serve test checks in the completion gate

@Darryl233

Darryl233 commented Mar 12, 2026

Copy link
Copy Markdown
Collaborator

I recommend you use AI to generate a more detailed PR description.

@zihugithub zihugithub changed the title [Benchmark] Add benchmark functional tests for DeepSeek, Aquila and Mixtral [Benchmark] Add benchmark functional test framework with Qwen3 test case Mar 13, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Different chips performance differently, so maybe it's better to distinguish the gold values of different chips

Darryl233
Darryl233 previously approved these changes Mar 18, 2026

@Darryl233 Darryl233 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Caozhou1995 Caozhou1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Darryl233 Darryl233 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Caozhou1995 Caozhou1995 merged commit 73d9beb into flagos-ai:main Mar 19, 2026
71 of 114 checks passed
LiJunscs pushed a commit to LiJunscs/FlagScale that referenced this pull request Apr 13, 2026
…ase (flagos-ai#1145)

### PR Category
CICD

### PR Types
New Features

### PR Description

Add a benchmark functional test framework to the CI/CD pipeline,
enabling automated detection of training performance regressions by
comparing runtime metrics against baseline gold values.

- Introduce dedicated benchmark workflow
(`functional_tests_benchmark.yml`) with artifact download retry logic,
environment auto-setup, benchmark log parsing, metrics upload to
backend, and failure log archiving
- Add `test_benchmark_equal` in `check_results.py` supporting two
threshold types (`upper_bound` for elapsed time, `lower_bound` for
throughput) with configurable tolerance and automatic warmup iteration
skipping (first 5 iterations)
- Add `parse_benchmark_output.py` to extract metrics from training logs
into structured JSON and upload to the metrics backend for tracking
- Register `benchmark` as a new task type in the test runner
(`run_functional_tests.sh`), platform config loader
(`load_platform_config.sh`), and platform definition (`cuda.yaml`)
- Add Qwen3 tp2_pp2 as the first benchmark test case with gold values
for elapsed time per iteration (ms) and throughput per GPU (TFLOP/s/GPU)
at 10% tolerance
- Integrate benchmark tests into `all_tests_common.yml` and re-enable
inference/serve test checks in the completion gate

---------

Co-authored-by: zihugithub <fbye@baai.ac.cn>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants